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ABSTRACT 

U.S.  Environmental  Protection  Agency  (EPA)  Emission 
Measurement  Center  in  conjunction  with  EPA  Regions  VI 
and  VIII,  the  state  of  Utah,  and  the  U.S.  Department  of 
Defense  have  conducted  a  series  of  long-term  pilot  and 
field  tests  to  determine  the  accuracy  and  reliability  of  a 
visible  opacity  monitoring  system  consisting  of  a  conven¬ 
tional  digital  camera  and  a  separate  computer  software 
application  for  plume  opacity  determination.  This  tech¬ 
nology,  known  as  the  Digital  Opacity  Compliance  System 
(DOCS),  has  been  successfully  demonstrated  at  EPA-spon- 
sored  Method-9  "smoke  schools/'  as  well  as  at  a  number 
of  government  and  commercially  operated  industrial  fa¬ 
cilities. 

Results  from  the  current  DOCS  regulatory  pilot  study 
demonstrated  that,  under  regulatory  enforcement  condi¬ 
tions,  the  average  difference  in  opacity  measurement  be¬ 
tween  the  DOCS  technology  and  EPA  Reference  Method  9 
(Method  9)  was  1.12%.  This  opacity  difference,  which  was 
computed  from  the  evaluation  of  241  regulated  air 
sources,  was  found  to  be  statistically  significant  at  the 
99%  confidence  level.  In  evaluating  only  those  sources  for 
which  a  nonzero  visible  opacity  level  was  recorded,  the 


IMPLICATIONS 

Results  from  the  DOCS  pilot  study  demonstrated  that,  un¬ 
der  regulatory  enforcement  conditions,  the  average  differ¬ 
ence  in  opacity  measurement  between  the  DOCS  technol¬ 
ogy  and  Method  9  certified  human  observers  was  1.12%. 
This  opacity  difference,  which  was  computed  from  the  eval¬ 
uation  of  241  regulated  air  sources,  was  found  to  be  sta¬ 
tistically  significant  at  the  99%  confidence  level.  Because  of 
the  successful  deployment  of  DOCS  under  regulatory  en¬ 
forcement  conditions,  EPA  has  recently  developed  a  cam¬ 
era-based  visible  opacity  measurement  approach  titled 
“Determination  of  Visible  Emission  Opacity  from  Stationary 
Sources  Using  Computer-Based  Photographic  Analysis 
Systems”  (available  at  http://www.epa.gov/ttn/emc/prelim/ 
pre-008.pdf).  The  proposed  opacity  measurement  ap¬ 
proach  is  scheduled  to  be  promulgated  by  EPA  in  2006  as 
a  conditional  method  after  completion  of  the  public  com¬ 
ment  period. 


average  difference  in  opacity  measurement  between  the 
DOCS  technology  and  Method  9  was  1.20%.  These  results 
suggest  that  the  two  opacity  measurement  methods  are 
essentially  equivalent  when  measuring  the  opacity  of  vis¬ 
ible  emissions. 

Given  the  costs  and  technical  limitations  associated 
with  use  of  Method  9,  there  is  a  recognized  need  to  de¬ 
velop  accurate,  reproducible,  and  scientifically  defensible 
alternatives  to  the  use  of  human  observers.  The  use  of 
digital  imaging/processing  brings  current  technology  to 
bear  on  this  important  regulatory  issue.  Digital  technol¬ 
ogy  offers  increased  accuracy,  a  permanent  record  of  mea¬ 
surement  events,  lower  costs,  and  a  scientifically  defensi¬ 
ble  approach  for  opacity  determination. 

INTRODUCTION 

Most  U.S.  Department  of  Defense  (DoD)  installations/ 
facilities  are  subject  to  Title  V  of  the  1990  Clean  Air  Act 
Amendments.  Although  there  are  a  variety  of  air  sources 
regulated  under  Title  V,  the  most  common  are  those  that 
generate  visible  emissions.1  To  demonstrate  compliance 
with  federal  visible  emission  limits,  the  opacity  associated 
with  regulated  air  sources  must  be  verified  through  use  of 
an  approved  regulatory  method. 

For  the  majority  of  regulated  air  sources,  the  primary 
method  for  determining  compliance  with  permitted 
opacity  levels  is  U.S.  Environmental  Protection  Agency 
(EPA)  Reference  Method  9  (Method  9). 2  Method  9  relies 
on  trained  human  observers  to  visually  determine  com¬ 
pliance  by  measuring  the  opacity  of  a  smoke  plume  once 
every  15  sec  for  a  specified  time  period.  The  15-sec  opacity 
recordings  are  averaged  to  determine  a  single  opacity 
measurement  that  is  then  compared  against  the  permit¬ 
ted  opacity  level  of  the  facility. 

To  become  legally  certified  as  a  Method  9  visual  opac¬ 
ity  observer,  an  individual  must  complete  classroom 
training  and  successfully  pass  a  visual  opacity  field  exam¬ 
ination  conducted  at  an  EPA-approved  smoke  school 
once  every  6  months.  The  field  examination  requires  that 
the  Method  9  candidate  determine  the  visible  opacity  of 
25  white  and  25  black  smoke  plumes  with  an  error  rate  of 
<15%  for  any  individual  opacity  observation  and  an  ag¬ 
gregate  opacity  measurement  error  rate  of  <7.5%  for  all 
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50  readings.  Although  Method  9  has  an  extensive  history 
of  successful  use,  its  opacity  measurements  are  inherently 
subjective,  a  characteristic  that  renders  its  results  vulner¬ 
able  to  claims  of  inaccuracy,  bias,  and,  in  some  cases, 
outright  fraud. 

Beyond  the  technical  concerns  associated  with  the 
limited  accuracy  and  reliability  of  Method  9  results,  the 
recurring  training  costs  necessary  to  support  Method  9 
certification  can  become  a  significant  financial  burden  on 
the  compliance  budget  of  a  facility.  With  historically  flat 
and  currently  declining  federal  environmental  compli¬ 
ance  budgets,  development  of  new  cost-effective  and 
regulatorially  supportable  methods  for  verifying  compli¬ 
ance  with  permitted  visible  opacity  levels  is  receiving 
increased  attention  by  environmental  compliance  en¬ 
forcement  personnel,  as  well  as  the  regulated  community.3 

The  Digital  Opacity  Compliance  System  (DOCS), 
which  is  an  innovative  technology  that  uses  digital  imag¬ 
ing  technology  for  quantifying  visible  opacity,  has  been 
developed  and  field  tested  as  a  technically  defensible  and 
economically  competitive  method  for  verifying  compli¬ 
ance  with  permitted  opacity  levels.4-6  The  DOCS  uses  a 
commercial-off-the-shelf  digital  camera  to  capture  images 
of  visible  opacity,  which  are  then  downloaded  to  a  stan¬ 
dard  personal  computer  and  analyzed  using  specialized 
computer  software.  The  DOCS  technology  has  been  dem¬ 
onstrated  to  be  not  only  as  accurate  and  reliable  as 
Method  9  but  has  the  added  advantage  of  furnishing  the 
technology  user  with  a  permanent  visual  record  of  the 
emissions. 

BACKGROUND 

The  presence  of  visible  air  emissions  from  regulated  sta¬ 
tionary  air  sources  provides  irrefutable  evidence  that  air¬ 
borne  particles  are  being  discharged  into  the  atmo¬ 
sphere.17  The  public  has  not  only  expressed  its  concern 
regarding  the  negative  psychological  effects  of  visible 
emissions,  but,  under  many  circumstances,  particulate 
emissions  have  been  identified  as  the  cause  of  increased 
human  health  and  environmental  risk.  Because  of  public 
concerns  regarding  the  potential  health  and  environmen¬ 
tal  impacts  associated  with  visible  air  emissions,  current 
federal  statutes,  as  well  as  many  state  and  local  air  quality 
control  laws,  actively  regulate  the  opacity  of  plumes  (i.e., 
point  sources  of  air  pollution). 

The  current  technology  field  demonstration  was  de¬ 
signed  to  evaluate  the  technical  performance  of  the  DOCS 
technology  under  regulatorially  enforceable  conditions. 
The  implementation  of  the  field  demonstration,  which 
was  formulated  to  complement  previous  DOCS  field  ac¬ 
tivities,  was  requested  by  EPA  Emission  Measurement 
Center  (EMC)  to  establish  the  technical  basis  for  a  new 
regulatorially  approved  method  for  measuring  visible 
opacity.3'8  Independent  regulatory  and  technical  over¬ 
sight  for  the  DOCS  1-yr  regulatory  pilot  demonstration 
study  was  provided  by  a  DOCS  regulatory  and  technical 
advisory  panel,  which  consisted  of  EPA  scientists,  engi¬ 
neers,  and  air  quality  enforcement  personnel  from  Re¬ 
gions  VI  (Texas)  and  VIII  (Colorado),  as  well  as  state 
regulators  from  California,  Texas,  and  Utah.  Beyond  the 
regulatory  community,  DoD  air  compliance  personnel 
from  Elmendorf  Air  Force  Base,  AK,  Fort  Hood,  TX,  Hill 


Figure  1.  Drawing  of  the  opacity  analysis  box  using  the  DOCS 
technology. 


Air  Force  Base,  UT,  and  Aberdeen  Proving  Ground,  MD, 
participated  in  the  DOCS  regulatory  and  technical  advi¬ 
sory  panel  deliberations. 

Basic  Theory  of  the  DOCS  Technology 

During  the  application  of  the  DOCS  technology,  digital 
photographs  of  visible  emissions  are  taken  from  positions 
that  allow  the  plume  to  be  clearly  viewed  against  a  sky 
background.  These  photographs  are  downloaded  to  a  lap¬ 
top  or  desktop  computer  where  they  can  be  analyzed  for 
opacity  using  the  DOCS  software.  The  initial  steps  in 
analyzing  the  digital  image  for  opacity  include:  (1)  acti¬ 
vating  the  DOCS  opacity  analysis  program,  (2)  retrieving 
those  digital  photographs  that  are  to  be  evaluated,  and  (3) 
using  the  DOCS  program  to  draw  an  analysis  box  (or  grid) 
around  that  portion  of  the  visible  emissions  that  will  be 
analyzed  for  opacity  (Figure  1). 

After  selection  of  the  analysis  box,  the  DOCS  software 
first  distinguishes  whether  the  plume  is  lighter  or  darker 
than  the  background.  The  DOCS  software  accomplishes 
this  by  transforming  the  image  to  a  gray  scale  with  en¬ 
hanced  dark  and  light  colors.  By  assuming  that  the  spatial 
color  intensity  of  the  pixels  located  at  the  vertical  edges  of 
the  analysis  box  correspond  with  background  and  those 
located  in  the  center  represent  visible  emissions,  the  soft¬ 
ware  is  able  to  determine  statistically  whether  the  emis¬ 
sions  are  lighter  or  darker  than  the  background. 

Once  the  contrast  between  the  emissions  and  back¬ 
ground  has  been  established,  the  DOCS  software  trans¬ 
forms  the  spatial  color  intensity  data  (which  is  stored  in 
image  files)  to  an  optical  space  characterized  by  the  di¬ 
mensions  of  hue,  saturation,  and  value  (HSV).  In  HSV 
space,  the  DOCS  software  uses  a  principal  component 
analysis  approach  to  identify  the  most  important  param¬ 
eter  gradient  (i.e.,  the  parameter  gradient  that  reflects  the 
maximum  variability  in  color  intensity).  That  parameter 
(or  principal  component)  is  then  used  to  establish  a  linear 
scale  of  opacity  from  which  an  overall  aggregate  opacity 
of  the  plume  can  be  determined.9 

Practical  Application  of  the  DOCS  Technology 

Under  normal  circumstances,  the  DOCS  technology  user 
rarely  needs  to  understand  the  complex  mathematical 
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relationships  associated  with  digital  image  transforma¬ 
tion.  Rather,  the  technology  user  simply  draws  an  analysis 
box  around  the  area  of  the  plume  to  be  analyzed  for 
opacity.  The  DOCS  software  then  establishes  the  plume 
background  based  on  the  spectral  information  contained 
in  the  pixels.  The  size  and  shape  of  the  analysis  box, 
which  is  controlled  by  the  user  of  the  DOCS  software, 
must  be  chosen  judiciously,  because  the  final  opacity 
measurement  will  ultimately  depend  on  what  part  of  the 
image  the  DOCS  software  identifies  as  background. 

A  compelling  advantage  of  DOCS  over  other  pro¬ 
posed  camera-based  opacity  determination  systems  is  that 
its  application  does  not  require  the  use  of  physical  targets 
to  achieve  accurate  and  reproducible  opacity  measure¬ 
ments.  In  principle,  the  use  of  targets  for  visible  ob- 
scurence  measurement  would  seem  to  have  scientific  mer¬ 
it;  however,  in  practice,  the  economic  burden  associated 
with  target  maintenance  and  logistical  operations  (partic¬ 
ularly  for  those  targets  placed  on  stacks  of  considerable 
height,  i.e.,  >15  ft)  have  never  been  satisfactorily  ad¬ 
dressed  and,  in  some  cases,  could  be  significant.14'15 

Finally,  the  most  persuasive  argument  supporting  a 
nontarget-based  digital  camera  approach  to  determining 
stack  plume  opacity  is  the  availability  of  Method  9.  Be¬ 
cause  Method  9  has  been  successfully  litigated  by  both 
federal  and  state  regulatory  agencies,  both  regulated  facil¬ 
ities  and  the  regulating  community  are  inherently  averse 
to  using  any  technology  deemed  less  user  friendly  and/or 
more  costly  to  implement  than  Method  9.  The  DOCS 
technology,  which  is  essentially  a  "point  and  shoot"  pho¬ 
tographic  approach,  is  as  easy  to  apply  as  Method  9  in  the 
field,  while  providing  the  user  with  improved  compliance 
documentation,  as  well  as  enhanced  objectivity  and  re¬ 
producibility  in  opacity  determination. 

Previous  DOCS  Technology  Field  Demonstrations 

From  January  2001  through  December  2003,  a  DOCS  field 
demonstration  was  conducted  during  which  the  digital 
camera-based  opacity  measurement  technology  was  eval¬ 
uated  at  EPA-approved  Method  9  smoke  schools,  as  well 
as  DoD  industrial  and  private  commercial  facilities.3-8 
Results  from  these  field  tests  confirmed  that  the  DOCS 
technology  consistently  met  the  quantitative  perfor¬ 
mance  standards  for  accuracy  and  reliability  established 
for  Method  9.  Furthermore,  over  the  opacity  ranges  of 
regulatory  importance  (i.e.,  0-40%  opacity),  the  DOCS 
accuracy  was  demonstrated  to  be  statistically  better  than 
that  achieved  by  Method  9-certified  human  observers. 
During  deployment  of  the  DOCS  technology  at  DoD  in¬ 
dustrial  and  private  commercial  facilities,  the  DOCS  ac¬ 
curacy  in  measuring  visual  opacity  was  found  to  not  only 
be  comparable  to  Method  9-certified  human  observers, 
but  the  DOCS  technology  exhibited  several  important 
practical  advantages  for  DoD  facilities  including:  (1)  im¬ 
proved  measurement  objectivity  and  reliability,  (2)  lower 
deployment  and  maintenance  costs,  and  (3)  generation  of 
a  permanent  digital  image  of  visible  opacity  that  can  be 
easily  referenced  in  challenging  regulatory  enforcement 
actions. 

After  a  comprehensive  technical  review  of  the  DOCS 
technology  field  demonstration  results  and  in  light  of 
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continuous  appeals  by  EPA  regional  offices,  state  regula¬ 
tory  agencies,  and  the  regulated  community  for  the  estab¬ 
lishment  of  a  camera-based  visible  opacity  field  measure¬ 
ment  method,  EPA  EMC  recommended  the  development 
and  implementation  of  a  DOCS  1-yr  regulatory  pilot 
study  in  which  the  accuracy  and  reliability  of  the  DOCS 
technology  would  be  evaluated  side-by-side  with  Method 
9-certified  human  observers  under  regulatory  enforce¬ 
ment  conditions.  The  DOCS  1-yr  regulatory  pilot  demon¬ 
stration  study  was  designed  to  collect  DOCS  technology 
performance  data  during  the  air  quality  enforcement  in¬ 
spection  of  a  range  of  regulated  air  sources  associated  with 
government  and  private  industrial  facilities. 

Goal  and  Objectives 

The  overarching  goal  of  the  DOCS  1-yr  regulatory  pilot 
study  was  to  achieve  regulatory  approval  for  the  use  of  the 
DOCS  technology  to  verify  compliance  with  permitted 
visible  opacity  limits  of  stationary  air  sources.  The  field 
demonstration  sites  selected  to  support  the  DOCS  1-yr 
regulatory  pilot  study  included  regulated  stationary  air 
sources  located  at  Fort  Wainwright,  AK,  Hill  Air  Force 
Base,  UT,  Fort  Hood,  TX,  and  a  number  of  public  and 
private  commercial  facilities  within  the  state  of  Utah.  The 
specific  DOCS  technology  demonstration  objectives  are 
summarized  as  follows:  (1)  collect  and  compile  all  opacity 
field  data  recorded  by  the  DOCS  technology,  as  well  as 
those  readings  taken  by  Method  9-certified  human  ob¬ 
servers  under  regulatory  enforcement  conditions;  (2)  us¬ 
ing  standard  statistical  significance  testing,  evaluate  the 
performance  of  the  DOCS  technology  relative  to  Method 
9-certified  human  observers  under  regulatory  enforce¬ 
ment  conditions;  and  (3)  based  on  the  DOCS  field  dem¬ 
onstration  results,  develop  a  draft  camera-based  visible 
opacity  measurement  method  for  review  and  consider¬ 
ation  by  EPA. 

EXPERIMENTAL  WORK 

The  DOCS  1-yr  regulatory  pilot  field  demonstration  study 
was  designed  to  estimate  the  visible  opacity  of  regulated 
stationary  air  sources  during  Method  9  compliance  veri¬ 
fication  activities  at  DoD  installations  located  in  the  states 
of  Alaska,  Texas,  and  Utah,  as  well  a  number  of  public  and 
private  commercial  facilities  within  the  state  of  Utah. 
During  DOCS  technology  field  testing,  Method  9-certified 
observers  determined  the  visible  opacity  associated  with 
specific  regulated  sources  once  every  15  sec  for  a  6-min 
period.  The  24  visible  opacity  measurements  were  then 
averaged  and  reported  as  the  Method  9  opacity  level  for 
the  specific  air  source. 

While  Method  9  field  measurements  were  being  con¬ 
ducted,  digital  photographs  of  the  visible  emissions  were 
taken  at  the  frequency  of  one  photograph  every  20  sec  for 
6  min.  At  the  end  of  the  6-min  measurement  period,  the 
18  digital  photographs  were  evaluated  using  the  DOCS 
analysis  software  by  personnel  from  the  participating  or¬ 
ganizations  and  the  results  averaged  to  generate  a  DOCS 
technology  opacity  measurement. 

At  U.S.  military  installations,  DoD  air  quality  compli¬ 
ance  inspectors  were  responsible  for  collecting  both  the 
Method  9  and  DOCS  technology  opacity  measurements. 
Similarly,  at  the  public  and  private  commercial  facilities 
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within  the  state  of  Utah,  inspectors  from  the  Utah  Divi¬ 
sion  of  Air  Quality  were  responsible  for  determining  vis¬ 
ible  opacity  levels  using  both  the  DOCS  technology  and 
Method  9.  All  of  the  field  demonstration  data  collected 
over  the  12-month  study  (including  opacity  determina¬ 
tions  and  digital  photographs)  were  furnished  to  a  DOCS 
quality  assurance  team,  which  had  the  responsibility  of 
compiling  the  opacity  data  for  statistical  analysis. 

DOCS  Regulatory  and  Technical  Advisory  Panel 

Before  collection  of  any  field  data,  a  DOCS  regulatory  and 
technical  advisory  panel  consisting  of  experienced  EPA 
scientists  and  engineers  and  federal  and  state  air  quality 
regulatory  personnel,  as  well  as  a  number  of  DoD  engi¬ 
neers,  scientists,  and  compliance  personnel  was  consti¬ 
tuted  to  review  the  scope  of  the  DOCS  1-yr  regulatory 
pilot  field  demonstration  study.  The  role  of  the  advisory 
panel  was  to  provide  regulatory  and  technical  review  and 
comment  on  the  DOCS  technology  field  demonstration 
protocol,  statistical  analysis  methods,  demonstration 
study  conclusions,  and  recommendations. 

After  completion  of  the  DOCS  field  demonstration 
data  collection  activities,  the  measured  opacity  of  each 
stationary  source  as  analyzed  by  the  DOCS  technology,  as 
well  as  by  Method  9-certified  human  observers,  was  fur¬ 
nished  to  the  DOCS  quality  assurance  officer  by  the  par¬ 
ticipating  organizations  on  a  monthly  basis  and  stored  in 
a  Microsoft  ACCESS  database.  An  assigned  quality  assur¬ 
ance  officer  served  as  the  central  repository  for  all  of  the 
DOCS  technolog>7  and  Method  9  field  data  collection, 
compilation,  and  quality  assurance  information.  At  the 
request  of  EPA,  monthly  conference  calls  were  scheduled 
for  the  DOCS  regulatory  and  technical  advisory  panel  to 
review  and  discuss  results,  as  well  as  to  modify  field  data 
collection  procedures,  when  necessary. 

Statistical  Test  Methods 

To  establish  credible  and  scientifically  defensible  argu¬ 
ments  that  support  approval  of  a  new  visible  opacity  test 
method,  data  from  the  DOCS  1-yr  regulatory  pilot  study 
were  compiled  and  analyzed  using  standard  statistical 
procedures.  During  the  field  data  collection  activities,  the 
visible  opacity  measurements  reported  by  the  DOCS  tech¬ 
nology  were  compared  against  Method  9-certified  visible 
observers  or  an  EPA-certified  transmissometer.  By  design¬ 
ing  the  data  collection  activities  to  result  in  paired  (or 
dependent)  visible  opacity  measurements,  the  impact  of 
factors  that  might  contribute  to  an  observed  difference  in 
performance  between  opacity  measurement  approaches 
(e.g.,  weather  conditions)  was  minimized. 

In  significance  testing,  a  null  hypothesis  (H0)  was 
developed  that  will  be  assumed  to  be  true  in  the  absence 
of  strong  quantitative  evidence  to  the  contrary.  The 
strength  of  the  data  may  be  evaluated  statistically  using 
either  a  paired  sample  t  test  or  by  constructing  a  confi¬ 
dence  interval  about  the  mean  difference  between  the 
two  methods.  The  results  of  the  paired  sample  t  test 
and/or  the  range  of  the  confidence  interval  will  provide 
the  basis  for  either  rejecting  or  not  rejecting  the  null 
hypothesis,  H0. 

H0  for  the  present  study  may  be  stated  as  follows: 
"the  true  mean  difference  between  Method  9  readings 


and  those  reported  by  the  DOCS  technolog}7  is  zero."  This 
statement  reflects  the  decision-maker's  conclusion  that 
the  two  opacity  measurement  methods  are  equivalent. 
Similarly,  the  alternative  hypothesis  (Ha)  may  be  con¬ 
structed  as  follows:  "the  true  mean  difference  between 
Method  9  readings  and  those  reported  by  the  DOCS  tech¬ 
nology  is  not  zero."  If  the  strength  of  the  data  is  sufficient 
to  reject  the  null  hypothesis,  the  decision-maker  will  con¬ 
clude  that  Ha  is  true.  In  statistical  terms,  these  hypotheses 
can  be  presented  as  follows: 


H0:  80  =  0  and  Ha:  80  *  0  (1) 


where  80  is  the  true  mean  difference  between  opacity 
readings  made  by  Method  9  and  the  DOCS  technology. 

Because  the  true  mean  difference  between  the  two 
visible  opacity  measurement  methods  (50)  can  never  be 
known  exactly,  it  must  be  determined  by  calculating  the 
average  difference.  Equation  2  was  used  to  calculate  the 
average  of  the  paired  opacity  differences  from  the  sam¬ 
pling  data: 


1  =  n  ^  ^  i  =  n 

5=  X  jf  =  -‘  2  (Yu-Yza)  (2) 

i  =  1  i  =  1 


where  d  is  the  average  difference  between  paired  opacity 
measurements;  ylfi>  y2/i  is  the  opacity  measurement  i  re¬ 
corded  by  the  Method  9  observers  and  the  DOCS,  respec¬ 
tively;  and  n  is  the  number  of  paired  opacity  measure¬ 
ments. 

Equations  3  and  4  were  used  to  determine  the  sample 
variance  and  standard  error  of  the  average  differences 
between  opacity  readings,  respectively.  To  use  the  paired 
t  test  to  draw  defensible  conclusions  from  the  dataset 
requires  that  the  decision-maker  select  a  level  of  signifi¬ 
cance  (a)  from  which  a  critical  t  value  may  be  determined: 


2  (di-d)2 

o  i  =  1 

Variance :  s/  = - ; —  (3) 

a  n  -  1  v  ' 


Standard  Error :  sa  = 


j*d_ 


(4) 


where  dt  is  the  difference  between  paired  opacity  measure¬ 
ments. 

Given  an  assigned  level  of  significance,  a,  and  degrees 
of  freedom  (n  -  1),  the  critical  t  values  and  test  statistic 
( ttest )  are  defined  by  the  following  expressions,  which  are 
compared  to  determine  whether  the  strength  of  the  field 
data  is  sufficient  to  reject  the  null  hypothesis,  H0: 


Critical  t  values:  ta  .  and-f«  ,  (5) 

f.n-l  2'n  “  1 
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Table  1.  Statistical  significance  testing  of  DOCS  field  demonstration  data. 


Data  Type 

na 

Average  Difference11 
(%) 

Test 

Statistic0 

Critical  t 
Value" 

Rejection  of  Null 
Hypothesis 

All  data 

241 

-1.12 

-41.8 

2.576 

Yes 

All  observations  where  visual  opacity  was 
reported  to  be  >0  as  measured  by 

Method  9-certified  human  observers 

36 

1.20 

0.67 

2.704 

No 

Notes:  aNumber  of  regulated  air  sources  evaluated;  Average  difference  is  computed  based  on  the  following  equation:  opacity  level  (Method  9)  -  opacity  level 
(DOCS);  Computed  based  on  field  data;  dTaken  from  standard  t  tables  assuming  a  99%  confidence  level  (a  =  0.01). 


d  -  80 

Test  statistic :  £test  = -  (6) 

where  80  is  the  true  mean  difference  (assumed  equal  to 
zero).  Test  condition:  If  ttest  >  ta/2>  n-i  or  Uest  < 
-  ta/2r  n  _  lt  then  the  null  hypothesis,  H0,  is  rejected. 

A  method  statistically  equivalent  to  the  paired  t  test 
for  determining  whether  the  strength  of  the  data  is  suffi¬ 
cient  to  reject  the  null  hypothesis  involves  generating  a 
(1-a)  confidence  interval  about  the  average  difference 
using  eq  7.10'11  If  the  confidence  interval  contains  zero, 
the  conclusion  will  be  that,  at  the  stated  confidence  level 
(1— a),  the  true  mean  difference  between  the  two  opacity 
measurement  methods  (SD)  is  insignificant,  and,  there¬ 
fore,  the  strength  of  the  data  is  insufficient  to  reject  the 
null  hypothesis  Hc  (i.e.,  the  two  opacity  measurement 
methods  are  statistically  equivalent). 

Conversely,  if  the  confidence  interval  does  not  con¬ 
tain  zero,  the  conclusion  will  be  that,  at  the  stated  confi¬ 
dence  level  (1— a),  the  true  mean  difference  between  the 
opacity  readings  of  the  two  measurement  methods  (80)  is 
significant,  and,  therefore,  the  evidence  is  sufficient  to 
reject  the  null  hypothesis,  H0,  and  the  decision-maker 
will  accept  the  alternative  hypothesis,  Ha  (i.e.,  the  two 
opacity  measurement  methods  are  statistically  different), 
as  true. 

d±en-rsa  (7) 

where  ta/2,n  -  1  is  the  critical  t  value  from  (t  distribution 
tables);  a/2  is  the  tail  area  probability;  a  is  the  level  of 
significance  (i.e.,  for  99%  confidence,  a  =  0.01);  n  -  1  is 
is  the  degrees  of  freedom;  and  s3  is  the  standard  error  of 
the  average  differences  in  opacity  readings. 

RESULTS 

During  the  DOCS  1-yr  regulatory  pilot  study,  the  state  of 
Utah  Division  of  Air  Quality  in  conjunction  with  Fort 
Wainwright,  AK,  Hill  Air  Force  Base,  UT,  and  Fort  Hood, 
TX,  completed  opacity  measurements  of  241  regulated  air 
processes  using  both  Method  9-certified  smoke  readers 
and  the  DOCS  technology.  The  range  of  regulated  air 
processes  evaluated  by  the  two  methods  included:  (1) 
industrial  process  scrubbers,  (2)  coal-fired  boilers,  (3)  in¬ 
dustrial  air  strippers,  (4)  industrial  bag  houses,  (5)  emer¬ 
gency  power  generators,  (6)  asphalt  paving  operations,  (7) 


oil  refining,  (8)  chlorine  manufacturing,  (9)  steel  produc¬ 
tion,  (10)  meat  packing  operations,  (11)  incineration,  (12) 
gypsum  manufacturing,  (13)  gas/oil  distribution  systems, 
and  (14)  metal  coating  facilities. 

At  the  recommendation  of  the  DOCS  technical  and 
regulatory  advisory  panel,  decisions  resulting  from  these 
field  tests  were  to  be  supported  at  the  99%  confidence 
level;  therefore,  a  significance  level  (a)  of  0.01  was  used 
throughout  the  analysis.  Table  1  provides  a  summary  of 
the  statistical  analysis  of  the  DOCS  1-yr  regulatory  pilot 
field  demonstration  study.  It  should  be  noted  that,  in 
determining  the  average  difference  in  opacity  measure¬ 
ments  between  the  two  measurement  methods,  the  DOCS 
opacity  measurement  was  subtracted  from  the  Method  9 
opacity  observation.  Finally,  for  the  majority  of  regulated 
air  sources  evaluated  in  the  DOCS  field  demonstration 
study,  a  visible  opacity  level  of  zero  was  reported  by  the 
Method  9-certified  human  observers.  Table  1  provides  a 
statistical  assessment  of  the  two  measurement  approaches 
using  all  of  the  collected  field  data  (241  regulated  air 
sources),  as  well  as  only  those  regulated  air  sources  that 
reported  nonzero  visible  opacity  levels  (36  regulated  air 
sources). 

The  average  difference  in  opacity  readings  between 
the  two  methods  was  found  to  be  1.12%  when  all  of  the 
regulated  stationary  source  opacity  data  was  taken  into 
account.  Because  the  DOCS  opacity  measurement  was 
subtracted  from  the  Method  9-certified  human  observer 
opacity  reading,  this  finding  suggests  that  the  DOCS  tech¬ 
nology  reports  slightly  higher  opacity  levels  than  the 
Method  9-certified  human  observers.  When  the  dataset 
was  limited  to  only  those  regulated  stationary  sources  for 
which  a  nonzero  opacity  level  was  recorded  by  the 
Method  9-certified  human  observers,  the  average  differ¬ 
ence  in  opacity  readings  between  the  two  methods  was 
found  to  be  1.20%,  a  finding  that  indicates  that  Method 
9-certified  human  observers  reported  opacity  values  that 
were  slightly  greater  than  those  measured  by  the  DOCS 
technology. 

In  applying  statistical  significance  testing,  the  results 
summarized  in  Table  1  suggests  that,  when  the  opacity 
data  from  all  241  of  the  regulated  air  sources  are  taken 
into  account,  comparison  of  the  critical  t  value  with  the 
test  statistic  supports  the  rejection  of  the  null  hypothesis. 
This  finding  is  tantamount  to  concluding  that  the  two 
opacity  approaches  are  different  and  that,  on  average,  the 
DOCS  technology  measures  visible  opacity  at  a  level  that 
is  —1.12%  greater  than  the  visible  opacity  measured  by 
Method  9-certified  human  observers. 
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Table  2.  Evaluation  of  the  99%  confidence  interval  about  the  average  opacity  difference. 


Average  Difference11 

99%  Confidence 

Data  Type 

na 

(%) 

Interval 

All  data 

241 

-1.12 

-1.18  <  -1.12  <  -1.06 

All  observations  where  visual  opacity  was 

36 

1.20 

-1.77  <  1.20  <  4.16 

reported  to  be  >0as  measured  by  Method 

9-certified  human  observers 

Notes:  aNumber  of  regulated  air  sources  evaluated;  bAverage  difference  is  computed  based  on  the  following  equation:  opacity  level  (Method  9)  -  opacity  level 
(DOCS). 


In  contrast,  when  only  those  regulated  stationary  air 
sources  for  which  a  nonzero  visible  opacity  was  detected 
by  the  Method  9-certified  human  observers  are  used  in 
the  statistical  analyses,  comparison  of  the  critical  t  value 
with  the  test  statistic  does  not  support  rejection  of  the 
null  hypothesis.  This  result  indicates  that,  when  measur¬ 
ing  nonzero  opacity  levels,  the  accuracy  of  the  two  mea¬ 
surement  approaches  is  statistically  equivalent. 

Similar  statistically  supportable  conclusions  can  be 
drawn  by  evaluating  the  99%  confidence  interval  about 
the  average  difference  in  opacity  readings.  Table  2  sum¬ 
marizes  the  results  of  this  statistical  approach.  When  data 
from  all  241  of  the  regulated  stationary  air  sources  are 
taken  into  account,  the  99%  confidence  interval  about 
the  average  opacity  difference  extends  over  a  range  that 
does  not  include  the  value  of  zero.  In  practical  terms,  this 
finding  supports  the  conclusion  that  the  two  methods  are 
statistically  different  and  that  the  DOCS  technology  mea¬ 
sures  visible  opacity  at  levels  that  are,  on  average,  1.12% 
higher  than  those  reported  by  Method  9-certified  human 
observers. 

Conversely,  when  only  those  regulated  stationary  air 
sources  for  which  a  nonzero  visible  opacity  was  measured 
by  the  Method  9-certified  human  observers  are  included 
in  the  statistical  analysis,  the  99%  confidence  about  the 
average  difference  extends  over  a  range  that  does  include 
the  value  of  zero.  The  conclusion  drawn  from  this  finding 
is  that  the  two  opacity  measurement  methods  are  statis¬ 
tically  equivalent  or,  alternatively,  there  is  less  than  a  1% 
probability  that  the  two  methods  are  different.10'11 

DISCUSSION 

Although  in  previous  field  tests,  the  DOCS  technology 
had  been  demonstrated  to  be  an  accurate  and  reliable 
digital  camera-based  method  for  determining  the  visible 
opacity  associated  with  stationary  air  sources,  these  activ¬ 
ities  were  conducted  under  relatively  controlled  condi¬ 
tions.  Most  of  the  earlier  field  data  collection  was  con¬ 
ducted  at  either  an  EPA-approved  Method  9  smoke 
schools  or  at  DoD  industrial  or  commercial  sites  at  which 
plume  opacity  was  controlled  by  facility  personnel.12 

Recognizing  the  importance  of  documenting  the  per¬ 
formance  of  the  DOCS  technology  under  actual  compli¬ 
ance  enforcement  conditions,  EMC  in  collaboration  with 
several  EPA  regions  and  state  air  regulatory  agencies  re¬ 
quested  implementation  of  a  DOCS  1-yr  regulatory  pilot 
study.  The  data  collected  during  this  effort  would  be  used 
in  conjunction  with  performance  data  collected  in  earlier 
DOCS  technology  field  demonstrations  to  support  the 
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development  and  promulgation  of  a  new  EPA-approved 
visible  opacity  measurement  method.13 

The  average  opacity  difference  found  between  the 
DOCS  technology  and  the  Method  9-certified  human  ob¬ 
servers  was  found  to  be  1.12%.  Although  a  relatively  small 
difference,  statistical  analysis  of  the  field  data  confirmed 
that  the  opacity  difference  was  significant.  In  other 
words,  on  average,  the  DOCS  technology  would  be  ex¬ 
pected  to  measure  the  visible  opacity  at  a  level  that  is 
1.12%  greater  than  the  Method  9-certified  human  ob¬ 
server. 

Of  the  241  regulated  air  sources  evaluated,  the  ma¬ 
jority  (205)  of  air  sources  were  reported  to  have  emissions 
having  no  discernable  opacity  (as  recorded  by  Method 
9-certified  human  observers).  Because  of  the  importance 
of  characterizing  the  opacity  measurement  response  be¬ 
tween  the  Method  9-certifed  human  observers  and  the 
DOCS  technology  for  visible  air  emissions,  the  DOCS 
regulatory  and  technical  advisory  panel  recommended 
that  a  separate  statistical  comparison  be  made  of  those  air 
sources  with  reported  nonzero  visible  opacity. 

Analysis  of  the  36  regulated  air  sources  for  which  a 
nonzero  visible  opacity  level  was  reported  by  the  Method 
9-certified  human  observers,  the  average  opacity  differ¬ 
ence  found  between  the  DOCS  technology  and  the 
Method  9-certified  human  observers  was  1.20%.  More¬ 
over,  based  on  the  results  from  statistical  significance 
testing,  this  difference  was  found  to  be  insignificant.  In 
other  words,  the  small  difference  found  in  the  average 
opacity  measurements  between  the  DOCS  technology 
and  the  Method  9-certified  human  observers  could  be 
attributed  to  random  variability  (or  error)  and  that  the 
two  measurement  approaches  were  statistically  equiva¬ 
lent. 

Given  the  successful  field  demonstration  of  the  DOCS 
technology  under  regulatory  enforcement  conditions,  as 
well  as  the  identification  and  field  validation  of  commer¬ 
cially  available  digital  camera  systems  that  can  support 
the  DOCS  technology,  EMC  in  conjunction  with  DoD  has 
developed  a  new  digital  camera-based  visible  opacity  mea¬ 
surement  method  that,  under  certain  circumstances,  may 
be  used  in  lieu  of  Method  9. 13  The  draft  camera-based 
visible  opacity  measurement  method,  which  is  currently 
undergoing  public  comment,  is  currently  on  the  EPA  web¬ 
site. 

CONCLUSIONS 

The  DOCS  1-yr  regulatory  pilot  study  was  formulated  to 
answer  critical  questions  regarding  the  performance  of 
n  $ 
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the  digital  camera-based  opacity  measurement  system  un¬ 
der  regulatory  enforcement  conditions.  Based  on  the  sta¬ 
tistical  analysis  of  the  technology  demonstration  dataset; 
the  following  conclusions  were  supported:  (1)  over  the 
range  of  visible  opacity  of  regulatory  interest  (i.e., 
0-40%);  the  accuracy  of  the  DOCS  technology  was  found 
to  be  statistically  equivalent  to  Method  9;  (2)  over  the 
range  of  visible  opacity  of  regulatory  interest  (i.e., 
0-40%);  the  DOCS  technology  measures  visible  opacity 
with  an  accuracy  that  was,  on  average,  1.12%  greater  than 
Method  9-certified  human  observers;  (3)  based  on  the 
successful  field  demonstration  of  the  DOCS  technology,  a 
draft  camera-based  visible  opacity  measurement  method 
was  developed  and  submitted  to  EPA  for  technical  and 
regulatory  review.13 

Finally,  after  review  of  the  draft  camera -based  visible 
opacity  measurement  method,  EPA  issued  a  preliminary 
camera-based  visible  opacity  measurement  approach  ti¬ 
tled  "Determination  of  Visible  Emission  Opacity  from 
Stationary  Sources  Using  Computer-Based  Photographic 
Analysis  Systems"  (available  at:  http://www.epa.gov/ttn/ 
emc/prelim/pre-008.pdf).  The  proposed  approach  is 
scheduled  to  be  promulgated  in  2006  as  a  conditional 
method  after  completion  of  the  public  comment  period. 
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